Designing Special Post-Processing Rules for SVM-Based Chinese Word Segmentation

We participated in the Third International Chinese Word Segmentation Bakeoff. Specifically, we evaluated our Chinese word segmenter NEUCipSeg in the close track, on all four corpora, namely Academis Sinica (AS), City University of Hong Kong (CITYU), Microsoft Research (MSRA), and University of Pennsylvania/University of Colorado (UPENN). Based on Support Vector Machines (SVMs), a basic segmenter is designed regarding Chinese word segmentation as a problem of character-based tagging. Moreover, we proposed postprocessing rules specially taking into account the properties of results brought out by the basic segmenter. Our system achieved good ranks in all four corpora. 1 SVM-based Chinese Word Segmenter We built out segmentation system following (Xue and Shen, 2003), regarding Chinese word segmentation as a problem of character-based tagging. Instead of Maximum Entropy, we utilized Support Vector Machines as an alternate. SVMs are a state-of-the-art learning algorithm, owing their success mainly to the ability in control of generalization error upper-bound, and the smooth integration with kernel methods. See details in (Vapnik, 1995). We adopted svm-light1 as the specific implementation of the model. 1.1 Problem Formalization By formalizing Chinese word segmentation into the problem of character-based tagging, we ashttp://svmlight.joachims.org/ signed each character to one and only one of the four classes: word-prefix, word-suffix, word-stem and single-character. For example, given a two-word sequence“东南亚 人”, the Chinese words for ”Southeast Asia(东 南亚) people(人) ”, the character “东”is assigned to the category word-prefix, indicating the beginning of a word;“南”is assigned to the category word-stem, indicating the middle position of a word; “亚”belongs to the category word-suffix, meaning the ending of a Chinese word; and last,“人”is assigned to the category single-character, indicating that the single character itself is a word. 1.2 Feature Templates We utilized four of the five basic feature templates suggested in (Low et al. , 2005), described as